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(57) Abstract 

A barrier mechanism provides a low-latency method of synchronizing all or some or the processing elements (PEs) in a massively 
parallel processing system. The barrier mechamsm is supported by one or more physical barrier synchronization circuits each receiving 
an input from every PE in the processing system. Each PE has an associated barrier synchronization register, in which each bit is used 
as an input to one of a plurality of logical barrier synchronisation circuits. The hardware supports both a conventional barrier function 
and an alternative eureka function. Each bit in the barrier synchronization registers can be programmed to perform as either barrier or 
eureka function, and all bits of the registers and each barrier synchronization circuits functions independently. Partitioning among PEs is 
accomplished by a barrier mask and interrupt register which enable certain of the bits in the barrier synchronization registers to a denned 
groups of PEs. Further partitioning is accomplished by providing bypass points in the physical barrier syrichrotuzaoon circuits to subdivide 
the physical barrier synchronization circuits into several types of PE barrier partitions of varying size and shape. The barrier mask and 
interrupt register and the bypass points are used in concert to accomplish flexible and scalable partitions corresponding to user-desired sizes 
and shapes with a latency several orders of magnitude faster than existing software implementations. 
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BARRIER SYNCHRONIZATION FOR DISTRIBUTED 
MEMORY MASSIVELY PARALLEL PROCESSING SYSTEMS 

5 

Field of t he Invention 

The present invention relates generally to massively parallel processing 
systems, and more particularly to a barrier synchronization mechanism for facilitating 
synchronization between multiple processors in such a system. 

10 

BafikgEfililld of the Invention 
Massively parallel processing (MPP) systems are computing systems 
comprised of hundreds or thousands of processing elements (PEs). While the power 
of a multiple instruction-multiple data (MIMD) computer system lies in its ability to 
15 execute independent threads of code simultaneously, the inherently asynchronous 
states of the PEs (with respect to each other) makes it difficult in such a system to 
enforce a deterministic order of events when necessary. Program sequences 
involving interaction between multiple PEs such as coordinated communication, 
sequential access to shared resources, controlled transitions between parallel regions, 
20 etc., may require synchronization of the PEs in order to assure proper execution. 

An important synchronization capability in any programming model is 
the barrier. Barriers are points placed in the code beyond which no processor 
participating in a computation may proceed before all processors have arrived. Since 
processors wait at a barrier until alerted that all PEs have arrived, the latency of the 
25 barrier mechanism can be very important. The latency of a barrier mechanism is the 
propagation time between when the last processor arrives at a barrier, and when all 
processors have been notified that the barrier has been satisfied. During this period 
of time, all PEs may be idle waiting for the barrier to be satisfied. Hence barriers 
are a serialization point in a parallel code. 
30 Barriers can be implemented entirely by software means, but software 

schemes are typically encumbered by long latencies and/or limited parallelism 
restricting how many processors can arrive at the barrier simultaneously without 
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artificial serialization. Because a barrier defines a serialization point in a program, it 
is important to keep the latency as low as possible. 

Hardware support for barriers, while addressing the latency problems 
associated with barriers implemented by purely software means, can have other 
5 shortcomings that limit the utility of the mechanism in a production computing 
system. Production computing systems demand that the barrier resource (like all 
resources) be partitionable among multiple users while maintaining protective 
boundaries between users. In addition, the barrier resource must be rich enough to 
allow division between the operating system and the user executing within the same 
10 partition. Provision must also be made for fault tolerance to insure the robust nature 
. of the barrier mechanism. 

Hardware mechanisms may also suffer from an inability to operate 
synchronously. This inability may require that a PE, upon discovering that a barrier 
has been satisfied (all PEs have arrived at that point in the program), wait until all 
15 PEs have discovered that the barrier has been reached before it may proceed through 
the next section of program code. The ability to operate synchronously enables the 
barrier mechanism to be immediately reusable without fear of race conditions. 

Hardware mechanisms may also require that a PE explicitly test a 
barrier flag to discover when the barrier condition has been satisfied. This can 
20 prevent a PE from accomplishing other useful work while the barrier remains 

unsatisfied, or force the programmer to include periodic tests of the barrier into the 
program in order to accomplish other useful work while a barrier is pending. This 
can limit the usefulness of a barrier mechanism when used as a means of terminating 
speculative parallel work (e.g., a database search) when the work has been completed 
25 (e.g. the searched-for item has been found). 

Summary of the Invention 

To overcome the above described shortcomings in the art and provide 
key system resources necessary for production computing, the present invention 
30 provides a hardware mechanism that facilitates barrier synchronization in a massively 



WO 95/16236 



PCT7US94/14067 



3 

parallel processing system. The present barrier mechanism provides a partitionable, 
low-latency, immediately reusable, robust mechanism which can be used to alert all 
PEs in a partition when all of the PEs in that partition have reached a designated 
point in the program. The mechanism permits explicit testing of barrier satisfaction, 

5 or can alternately interrupt the PEs when a barrier has been satisfied. The present 
invention also provides an alternate barrier mode, called a eureka, that satisfies a 
barrier when any one PE has reached a designated point in the program, providing 
the capability to terminate speculative parallel work. The present barrier mechanism 
provides multiple barrier resources to a partition to allow pipelining of barriers to 

10 hide the latency as well as offering raw latencies 2-3 orders of magnitude faster than 
software implementations. The barrier mechanism is also partitionable for 
multi-users. Barriers are used to bulk synchronize the processors in a partition 
between loops where producer/consumer hazards may exist, or control entry and exit 
between segments of a program. 

15 

Brief Description of the Drawings 
The foregoing and other objects, features and advantages of the 
invention, as well as the presently preferred embodiments thereof, will become 
apparent upon reading and understanding the following detailed description and 
20 accompanying drawings in which: 

FIGURE 1 shows a simplified block diagram of a representative MPP 
system with which the present barrier mechanism can be used; 

FIGURE 2 shows a simplified block diagram of a processing element 
(PE), including a processor, local memory and associated shell circuitry; 
25 FIGURE 3 shows the format of the barrier synchronization registers 

BSRO and BSR1; 

FIGURE 4 shows a simplified radix-2 barrier synchronization circuit; 
FIGURE 5 shows a simplified radix-4 barrier synchronization circuit; 
FIGURE 6 shows the format of the barrier synchronization function 
30 (BSFR) register; 
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FIGURE 7 shows the format of the barrier synchronization mask and 
interrupt (BSMI) register; 

FIGURE 8 shows the bypass points in a simplified radix-2 barrier 

synchronization circuit; 
5 FIGURE 9 shows the barrier synchronization circuit of FIGURE 8, 

with some of the bypass points redirected to the fanout block; 

FIGURES 10A-E show how a 512 PE system can be partitioned at 
each level of a barrier synchronization circuit; 

FIGURE 1 1 shows how the barrier synchronization mask and interrupt 
10 (BSMI) register is used in concert with the bypass points in the barrier 

synchronization circuits to achieve the desired shape and size of a particular 
partition; 

FIGURE 12 shows the format of the barrier timing register 

BARJTMG register, 
15 FIGURE 13 shows a block diagram of a bypass point; and 

FIGURE 14 shows the hardware state sequencing implementation for a 
single barrier bit. 

DETAILED DE SPRIPTION OF THE PREFERRED EMBODIMENT 
20 In the following detailed description of the preferred embodiment, 

reference is made to the accompanying drawings which form a part hereof, and in 
which is shown by way of illustration a specific embodiment in which the invention 
may be practiced. It is to be understood that the present detailed description is 
intended as exemplary only, and that other embodiments may be utilized and 
25 structural changes made without departing from the spirit and scope of the present 
invention. 

The preferred MPP system, for which the present invention provides 
hardware support for barrier synchronization, is a MIMD massively parallel 
multiprocessor with a physically distributed, globally addressable memory. A 
30 representative MPP system 100 is shown in FIGURE 1. The MPP system 100 
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contains hundreds or thousands of processors, each accompanied by a local memory 
and associated support circuitry. Each processor, local memory and support circuity 
component is called a processing element (PE). The PE's in the MPP system 100 are 
linked via an interconnect network. 
5 The preferred MPP system 100 has a physically distributed memory, 

wherein each processor has a fovored, low latency, high bandwidth path to a local 
memory, and a longer latency lower bandwidth access to the memory banks 
associated with other processors over the interconnect network. In the preferred 
embodiment, the interconnect network is comprised of a 3-dimensional torus which 
10 when connected creates a 3-dimensional matrix of PEs. The torus design has several 
advantages, including speed of information transfers and the ability to avoid bad 
communication links. The interconnect network is also scalable in all three 
dimensions. The interconnect network is described in more detail in the copending 
and commonly assigned U.S. Patent Application Serial Number 07/983,979, entitled 
15 "Direction Order Routing in Multiprocessing Systems", to Gregory M. Thorsen, filed 
November 30, 1992, which is incorporated herein by reference. 

FIGURE 2 shows a simplified block diagram of a PE 200. An 
individual PE includes a high-performance RISC (reduced instruction set computer) 
microprocessor 202. In the preferred MPP system, microprocessor 202 is the 
20 DECChip 21064-AA RISC microprocessor, available from Digital Equipment 

Corporation. Each PE is coupled to a local memory 204 that is a distributed portion 
of the globally-addressable system memory, and includes a shell of circuitry that 
implements synchronization and communication functions facilitating interactions 
between processors. 

25 The shell circuitry includes an interconnection network router 206, 

used to connect multiple PEs in the three-dimensional toroidal "fabric". The 
interconnection network carries all data communicated between PEs and memories 
that are not local. An address centrifuge and block transfer engine 208 in the PE 
shell circuitry permits asynchronous (i.e., independent of the local processor) 

30 movement of data between the local memory 204 and remote memories associated 
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with other PEs, such as block transfers, with flexible addressing modes that permit a 
high degree of control over the distribution of data between the distributed portions 
of the system memory. The address centrifuge and block transfer engine are 
described in detail in the copending and commonty assigned U.S. Patent Application 
5 entitled "RECURSIVE ADDRESS CENTRIFUGE FOR DISTRIBUTED MEMORY 
MASSIVELY PARALLEL PROCESSING SYSTEMS", filed on even date herewith 
to Fromm, which application is incorporated herein by reference. 

The shell circuitry also includes a data prefetch queue 210 which 
allows the processor 202 to directly initiate data movement between remote 
10 memories and the local processor in a way that can hide the access latency and 
permit multiple remote memory references to be outstanding. 

Synchronization circuits in the shell permit synchronization at various 
different levels of program or data granularity in order to best match the 
synchronization method that is "natural" for a given parallelization technique. At the 
15 finest granularity, data-level synchronization is facilitated by an atomic swap 

mechanism that permits the locking of data on an element-by-element basis. A more 
coarse grain data-level synchronization primitive is provided by a messaging facility, 
which permits a PE to send a packet of data to another PE and cause an interrupt 
upon message arrival, providing for the management of message queues and 
20 low-level messaging protocol in hardware. Control-level synchronization at the 
program loop level is provided by a large set of globally accessible 
fetch-and-increment registers that can be used to dynamically distribute work (in the 
form of iterations of a loop, for instance) among processors at run time. The present 
invention provides yet another form of control-level synchronization, barrier 
25 synchronization, which is useful to control transitions between major program blocks 
(i.e., between loops performing actions on the same data sets). 

Barrier Synchronization 

The present invention provides hardware support for barrier 
30 synchronization which results in a low-latency method of synchronizing all or a 
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portion of the PEs in an MPP system. The barrier synchronization mechanism of the 
present invention may be used to perform two types of synchronization: barrier and 
eureka. 

A barrier is a point in program code where, after reaching the barrier, 
5 a processor must wait until all other processors participating in the computation have 
also reached the barrier. After all of the processors reach the barrier, the processors 
continue issuing program code. 

A programmer may place a barrier in a program between distributed, 
parallel loops performing operations on the same data. By doing this, the 
10 programmer ensures that all of the processors associated with a loop finish the loop 
(all writes to shared data complete) before any of the processors continue with other 
program instructions (access the new values of the shared data). 

A eureka is a point in program code where a condition is established 
that only a single processor need satisfy, thereby causing all processors to proceed 
15 beyond the eureka point To participate in a eureka event all processors initialize 
the eureka barrier mechanism described herein and enable an interrupt, then proceed 
executing program code to solve for the eureka condition. As soon as any processor 
completes the computation, it triggers the eureka, thus causing an interrupt to all PEs. 
The interrupt indicates that the eureka has been satisfied and all PEs may continue 
20 beyond the eureka point. 

Eureka synchronization has several uses, including database searches. 
Using eureka synchronization, a programmer can stop a database search as soon as 
any processor succeeds in finding the desired data rather than waiting for all of the 
processors to exhaust the search. 

25 

Logical Barrier Synchronization Circuits 

The preferred barrier mechanism has 16 logical barrier synchronization 
circuits. Each PE in the MPP system has an input to each of the 16 logical barrier 
synchronization circuits. The multiple barrier synchronization circuits facilitate 
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partitioning between users and the operating system, as well as providing redundant 
resources for fault tolerance. 

The inputs to the 16 logical barrier synchronization circuits are 16 bits 
which are contained in two special registers. Each PE contains two 8-bit registers 
5 called barrier synchronization register 0 (BSRO) 300 and barrier synchronization 
register 1 (BSR1) 301. FIGURE 3 shows the format of BSRO 300 and BSR1 301. 
Each of the 16 bits which comprise BSRO 300 and BSR1 301 is an input to one of 
the 16 logical barrier synchronization circuits. Thus, each PE has an input to each of 
the 16 logical barrier synchronization circuits. 
10 Barrier synchronization register 0 (BSRO) 300 is an 8-bit, readable and 

writable, general access register. Preferably, BSRO 300 contains the eight least 
significant barrier bits for a PE. When read from, the value of BSRO 300 represents 
the value of bits 2' through 20 of BSRO 300. The remaining bits are not valid. 
Table 1 shows the bit format of BSRO 300 and describes each bit of the register. 
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Ta ble ! 

RSRO 300 Format 



Bits 


Name 


2° 


Barrier Bit 2° 


2' 


Barrier Bit 2' 


2 2 


Barrier Bit 2 2 


2 3 


Barrier Bit 2 3 


2 4 


Barrier Bit 2 4 


2 5 


Barrier Bit 2 5 


2 6 


Barrier Bit 2 5 


2 7 


Barrier Bit 2 7 


2 e3 -2 8 


These bits are not used. 



15 Barrier synchronization register 1 (BSR1) 301 is an 8-bit, readable and 

writable, privileged access register. BSR1 301 contains the eight most significant 
barrier bits for a PE. When read from, the value of BSR1 301 represents the value 
of bits 2' 5 through 2* of BSR1 301 and 2 7 through 2° of BSR0 300. The remaining 
bits are not valid. Table 2 shows the bit format of BSR1 301 and describes each bit 

20 of the register. 
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labia 

BSRl 301 Format 

Biis Name 

2 7 -2° These bits are not used; however, when 

BSRl 301 is read, these bits contain the 
value of bits 2 7 through 2° of BSRO 300. 

5 2 s Barrier Bit 2 s 

2 9 Barrier Bit 2 9 

2 10 Barrier Bit 2 10 

2" Barrier Bit 2 11 

2 n Barrier Bit 2 12 

10 2 13 Barrier Bit 2 13 

2 14 Barrier Bit 2 14 

2 15 Barrier Bit 2' 5 

2 63 -2 16 These bits are not used. 

15 

All 16 of the logical barrier synchronization circuits function 
identically and operate independently. An example of the operation of the barrier 
synchronization circuits will now be given, using bit 2 2 of BSRO 300 when it is used 
for barrier synchronization and for eureka synchronization as an example for 

20 purposes of illustration. 

It shall be understood that the remaining bits function in the same way 
as bit 2 2 in the following discussion. Each logical barrier synchronization circuit is 
implemented as an AND-tree and fanout-tree circuit. FIGURE 4 shows a barrier 
synchronization circuit 400 in a simplified MPP system using 2-input AND gates to 

25 form the AND-tree. For simplicity of illustration, the MPP system shown in 

FIGURE 4 contains only eight PEs. It shall be understood, however, that the present 
barrier mechanism and barrier synchronization circuits can be used with a system 
having any number of PEs. 



WO 95/16236 PCT/US94/14067 

11 

Because the network in FIGURE 4 is accommodating 8 processors, 
PE0-PE7, and the AND-tree is implemented with 2-input AND gates, log 2 8=3 levels 
of AND-tree are required to arrive and a final barrier bit representing the logical 
product of all the PEs' barrier bits. 

5 As a starting condition, before barrier synchronization begins, bit 2 2 of 

BSRO 300 is reset to 0 in all of the PEs. When a processor satisfies the barrier 
condition, that processor sets bit 2 2 of its associated BSRO 300 register to 1. This 
action sends a 1 to one of the AND gates in the first layer of the AND-tree. 

The first layer of the AND-tree shown in FIGURE 4 contains four 

10 AND gates. Each AND gate receives signals from two PEs. For example, one AND 
gate may receive signals from bit 2 2 of BSRO 300 in PE 0 and bit 2 2 of BSRO 300 in 
PE 1. When all of the processors reach the barrier point in the program code (satisfy 
the barrier) they have each set bit 2 2 of their associated BSRO 300 to 1, causing the 
output of each of the four AND gates in the first level of the AND-tree to switch to 

15 1. 

The second level of the AND-tree of FIGURE 4 contains two AND 
gates. Each AND gate receives signals from two of the AND gates in the first level 
of the AND tree. When the output of all of the AND gates in the first level of the 
AND-tree are 1, the output of both the AND gates in the second level of the 
20 AND-tree are 1. 

The third level of the AND-tree contains the final AND gate. This 
AND gate receives signals from both AND gates in the second level of the 
AND-tree. When the output of both AND gates in the second level of the AND-tree 
are 1 , the output of the final AND gate is 1 . The output of the final AND gate sends 
25 an input to the fanout-tree circuit. The fanout tree circuit is used to report the result 
of the AND-tree back to PEs. 

The first fanout block in the fanout tree receives a 1 from the single 
level 3 AND gate. After creating two copies of the 1, the first fanout block sends 
the Ps to the two fanout blocks in the second level of the fanout tree. 
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The two fanout blocks in the second level of the fanout tree each 
create two copies of the 1. The two fanout blocks in the second level of the fanout 
tree then sends the Vs to four fanout blocks in the first level of the fanout tree. 

The four fanout blocks in the first level of the fanout tree each create 
5 two copies of the 1. The fanout blocks in the first level of the fanout tree then send 
the Ts to the eight PEs. This signals all of the PEs in the system that all of the 
processors have reached the barrier and the processor in the PE may continue with 
other program instructions. Because the fanout tree is dependent on the AND-tree, 
the fanout tree will report that all of the PEs have reached the barrier only when 
10 each PE in the system has set bit 2 2 to 1. 

As will be described more fully below in connection with FIGURE 14, 
the barrier mechanism is designed to be immediately reusable. This means that as 
soon as a processor detects that the barrier bit has cleared (all processors have 
arrived at the barrier), it can immediately set the barrier bit again to announce its 
15 arrival at the next barrier if applicable. Doing so does not affect the notification of 
the prior barrier satisfaction to any other PE. 

Eureka Synchronization 

A logical barrier synchronization circuit 400 may also be used to 
20 perform eureka synchronization. One use of the eureka function is for the 

synchronization of a distributed, parallel data search. Once one of the PEs has 
located the targeted information, the successful PE can set a eureka bit to inform the 
other PEs involved that the search is completed. 

To enable the eureka function, each PE contains a register called the 
25 barrier synchronization function register (BSFR). FIGURE 6 shows the format of the 
BSFR and how it functions in conjunction with the BSRO 300 and BSR1 301 
registers. The barrier synchronization function register (BSFR) is preferably a 16-bit, 
write-only, system privileged register. The BSFR contains a 16-bit mask that 
indicates which bits of BSRO 300 and BSR1 301 are used for barrier synchronization 
30 and which bits are used for eureka synchronization. Bits 0-7 of the BSFR control 
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the function of BSRO 300 and bits 8-15 of the BSFR control BSR1 301 . If a bit of 
the BSFR is set to 1, the corresponding bit of BSRO 300 or BSR1 301 is used for 
eureka synchronization. If a bit of the BSFR is set to 0, the corresponding bit of 
BSRO 300 or BSR1 301 is used for barrier synchronization. Table 3 shows the bit 
5 format of BSFR and describes each bit of the register. 



BSFR Format 

Bits Name 

10 2 7 -2° These bits indicate which bits of BSRO 300 are used for 

eureka synchronization. For example, as shown in Figure 
6, when bit 2 2 of the BSFR is set to 1, bit 2 2 of BSRO 
300 is used for eureka synchronization. When bit 2 2 of 
the BSFR is set to 0, bit 2 2 of BSRO 300 is used for 
barrier synchronization. 

2 ,5 -2 8 These bits indicate which bits of BSR1 301 arc used for 

eureka synchronization. For example, as shown in 
FIGURE 6, when bit 2 n of the BSFR is set to 1, bit 2 13 
of BSR1 301 is used for eureka synchronization. When 
bit 2" of the BSFR is set to 0, bit 2 13 of BSR1 301 is 
used for barrier synchronization. 

2 63 -2 16 These bits are not used. 



15 Because each of the 16 logical barrier synchronization circuits operate 

completely independently of each other, any of the bits in each of the two barrier 
synchronization registers BSRO 300 and BSR1 301 for a particular PE can be 
programmed to function in the conventional barrier mode, or in eureka mode. As 
will be described below in connection with FIGURE 14, in eureka mode the output 

20 of the AND-tree is read directly by the PEs with no intervening synchronization 
logic such as the latch used in conventional barrier synchronization. 

Because in eureka mode the output of the fanout tree is read directly 
by the PEs ? the barrier hardware is not synchronous as it is in conventional barrier 
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mode, nor is a eureka bit immediately reusable. Using a barrier bit in eureka mode 
requires the use of a second bit programmed to function in barrier mode in order to 
prevent race conditions. The conventional barrier is used to synchronize the entry 
and exit of the PEs into and out of a eureka barrier. This implementation means that 

5 only half of the 16 barrier bits can be used for eureka synchronization. 

Reading BSRO 300 or BSR1 301 returns the current state of the 
AND-tree. This allows the AND-tree to be used as an OR tree by using negative 
logic. After all PEs initialize their barrier bit to a logical 1, the output of the 
AND-tree can be read as a one by all PEs. If any PE writes a zero to its barrier bit, 

10 the output of the AND-tree will read as a zero, performing the OR function. 

A typical eureka sequence begins with all processors initializing the 
eureka bit to a logical 1 and setting the conventional barrier bit to a logical 1 . When 
the barrier switches to a zero (meaning all processors have initialized their eureka bit 
and joined the barrier), the eureka, bit is "armed". The processors then begin the 

IS parallel data search or other activity that is to be terminated by the eureka. When a 
processor satisfies the termination conditions, it clears its eureka bit to alert the other 
processors and sets its conventional barrier bit to indicate it has observed the eureka 
event. As each of the other PEs detect that the eureka event has occurred (because 
the output of the eureka AND-tree drops to a logical 0), it sets its conventional 

20 barrier bit and waits for the barrier to complete. When all PEs have joined the final 
barrier, they may proceed to arm the next eureka. 

Servicing A Barrier 

In the preferred embodiment of the present invention, the processor 
25 monitors the output of the fanout circuitry using one of two mechanisms: 

periodically testing the barrier bit (such as with a continuous loop) or enabling a 
barrier interrupt. 

Continuing to use bit 2 2 as an example, in the first mechanism, after 
the processor sets bit 2 2 of BSRO 300 to 1, the processor may enter a loop that 
30 continuously checks the value of bit 2 2 of BSRO 300. After receiving a 1 from the 
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fanout circuitry, the support circuitry in the PE resets bit 2 2 of BSRO 300 to 0. 
Because the processor is regularly checking the value of bit 2 2 of BSRO 300, the 
processor may continue executing program instructions as soon as it is detected that 
bit 2 2 of BSRO 300 is reset to 0. 

5 In the barrier interrupt mechanism, after the processor satisfies the 

barrier and sets bit 2 2 of BSRO 300 to 1, the processor enables a barrier interrupt. 
The processor may then issue program instructions that are not associated with the 
barrier. After receiving a 1 from the fanout circuitry, the support circuitry in the PE 
resets bit 2 of BSRO 300 to 0 and sets the barrier interrupt to the processor. The 

10 barrier interrupt indicates to the processor that all of the processors have reached the 
. barrier, and causes the processor to suspend the unrelated activity and return to 
executing instructions associated with the barrier. The advantage of the barrier 
interrupt over continuous polling is the ability to perform other useful work while the 
other processors are approaching the barrier. 

15 In the preferred embodiment, the microprocessor enables the barrier 

hardware interrupt using a hardware interrupt enable register (HIER) in the 
microprocessor system control register. For more information on the HIER and the 
system control register, refer to the DECChip 21064-AA RISC Microprocessor 
Preliminary Data Sheet, available from Digital Equipment Corporation, which is 

20 incorporated herein by reference. The DEC microprocessor has 6 inputs for external 
hardware interrupts. These inputs appear in the HIRR (Hardware Interrupt Request 
Register) in bit positions 5-7 and 10-12. One of these six inputs is designated as the 
Barrier Interrupt Request bit. 

The HIRR inputs can be enabled or disabled by a mask bit located in 

25 the HIER (Hardware Interrupt Enable Register) internal to the microprocessor. For 
more information on the HIRR and HIER registers, refer to pages 3-26 through 3-29 
in the Digital Equipment Corporation publication: EV-3 AND EV-4 
SPECIFICATION Version 2.0 May 3, 1991, which is incorporated herein by 
reference. 
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Those skilled in the art are aware that all RISC microprocessors 
provide comparable facilities for allowing and controlling the direct sampling of 
external hardware interrupt signals, and that the present invention is not limited to 
use with the DEC RISC microprocessor described herein. 

5 The interrupt input to the microprocessor is asserted whenever any of 

the barrier bits selected by the BSMI (barrier synchronization mask and interrupt, 
discussed below) make the transition from a logical 1 to a logical 0, The interrupt 
input to the microprocessor is cleared by writing a bit in the system control register. 
To assure that no satisfied barrier events are missed, the correct programming 

10 sequence would be to clear the interrupt, then read BSRO 300 and BSR1 301 to 
determine which bit(s) have toggled. 

After the support circuity sets the barrier hardware interrupt, the 
processor must read the BSMI register (as described below) to determine if the 
interrupt was associated with BSRO 300 or BSR1 301. When the processor reads the 

15 value of the BSMI register, the support circuitry clears the interrupt associated with 
BSRO 300 and the interrupt associated with BSR1 301. 

If a barrier interrupt occurs while the processor is reading the BSMI 
register, the interrupt still occurs and is not cleared. The processor must then read 
the value of the BSMI register again to determine if the interrupt was associated with 

20 BSRO 300 or BSR1 301 and to clear the interrupts. 



Logical Partitions 

Not all of the PEs in a multiprocessing system may need to be part of 
a barrier or eureka synchronization process. Also, it is often desirable to have 

25 several partitions of PEs operational on different tasks simultaneously. To facilitate 
this, each PE also contains a barrier synchronization mask and interrupt (BSMI) 
register which is used to enable or disable a logical barrier synchronization circuit for 
a PE. The BSMI register contains a mask that indicates which bits of BSRO 300 and 
BSR1 301 are enabled for the PE, thus defining which partition(s) a PE is a member 

30 of, and defining partitioning among the PEs. 
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The 16 independent barrier synchronization circuits visible to the user 
in the banier registers can be assigned by the operating system to different groups or 
teams of processors. The BSMI allows the barrier bits in any given processor to be 
enabled or disabled. Alone, the BSMI permits the division of the barrier resource 
5 among up to 16 different partitions with arbitrary membership. Used in conjunction 
with the banier network partitioning capability described herein below, the BSMI 
allows the flexible subdivision of the barrier resource among a very large number of 
partitions in a scalable fashion. 

FIGURE 7 shows the format of the BSMI register. The BSMI is a 
10 16-bit, readable and writable, system privileged register. When written to, the BSMI 
register controls which bits in BSRO 300 and BSR1 301 are enabled for use by the 
PE. Bits 7-0 of BSMI enable the bits in BSRO 300, while bits 15-8 enable the bits 
in BSR1 301. If a bit of the BSMI register is set to 1, the corresponding bit of 
BSRO 300 or BSR1 301 is enabled. If a bit of the BSMI register is set to 0, the 
15 corresponding bit of BSRO 300 or BSR1 301 is disabled. 

The BSMI register has a different bit format when written to than it 
does when it is read from. Table 4 shows the bit format of the BSMI register when 
it is written to and describes each bit of the register. 

A disabled BSRO 300 or BSR1 301 bit appears to the logical product 
20 network to be always satisfied (a logical "1"). This permits the barrier to function 
normally for other processors whose barrier bit is enabled. Reading a disabled 
barrier synchronization register bit returns a logical M 0'\ Writing a disabled barrier 
synchronization register bit has no effect. 
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Table 4 

BMSI Register Write Format 

flits 

2^.2° These bits indicate which bits of 

BSRO 300 are enabled for use by the 
PE. For example, when bit 2 2 of the 
BSMI register is set to 1, as shown in 
FIGURE 7, bit 2 2 of BSRO 300 is 
enabled for use by the PE. When bit 
2 2 of BSRO 300 is disabled and 
cannot be used by the PE. 

2**_2 8 These bits indicate which bits of 

BSR1 301 are enabled for use by the 
PE. For example, when bit 2 13 of the 
BSMI is set to 1, as shown in 
FIGURE 7, bit 2 13 of BSR1 301 is 
enabled for use by the PE. When bit 
2 U of the BSMI is set to 0, bit 2' 3 of 
BSR1 301 is disabled and cannot be 
used by the PE. 

2«.2 16 These bits are not used. 



10 Table 5 shows the bit format of the BSMI register when it is read from and 

describes each bit of the register. When read from, bits 2 14 and 2 15 of the BSMI 
register provide the current state of the barrier interrupts from BSRO 300 and BSR1 
301, respectively. After being read the BSMI register is cleared. 
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Tab k 5 
BSMI Register Read Format 

Bits Name 

2 l3 -2° These bits are not valid. 

5 2 14 This bit reflects the current state of 

the barrier interrupt associated with 
bits 2' through 20 of BSRO 300. 

2 15 This bit reflects the current state of 

the 

barrier interrupt associated with bits 2 
through 28 ofBSRl 301. 

2 63 -2 16 These bits are not valid. 

10 Software may use the BSMI register to allow any set number of PEs 

to use one of the logical barrier synchronization circuits, and thus set up a partition 
among the PEs. For example, software may set bit 2 2 of the BSMI register to 1 in 
only four of the PEs in an MPP system. In this case, only the four PEs with bit 2 2 
of the BSMI register set to 1 may use the logical barrier synchronization circuit 

15 associated with bit 2 2 of BSRO 300. This creates a logical barrier partition among 
the four PEs. 

The BSMI and BSFR registers can be used in concert to arrive at 
several different barrier synchronization mechanisms. Table 6 shows the effect of 
one bit from the BSMI register and the BSFR on the corresponding bit in BSRO 300 
20 orBSR1301. 
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Physical Barrier Synchronization Circuits 

Although each of the 16 bits in the BSRO 300 and BSR1 301 registers 
in each PE represent an input to one of 16 logical barrier synchronization circuits, 
the preferred embodiment of the present barrier synchronization mechanism does not 
5 contain 16 physical barrier synchronization circuits. Instead, in the preferred 

implementation, the system contains 4 physical barrier synchronization circuits. The 
16 bits of BSRO 300 and BSR1 301 are time multiplexed into the four physical 
barrier synchronization circuits. 

One exemplary physical barrier synchronization circuit is shown in 
10 FIGURE 5. The preferred barrier network is implemented using a radix-4 AND-tree 
. and fanout tree. As shown, a 1024 PE system contains log4 1024 = 5 levels in a 
barrier synchronization circuit. 

Table 7 shows the input to each of the four physical barrier 
synchronization circuits during each clock period (CP). Four CPs are required for 
IS the physical barrier synchronization circuits to receive the input from all 16 bits in 
BSRO 300 and BSR1 301. 

The input registers to the logical barrier synchronization circuits (as 
shown and described below in connection with FIGURE 14) are completely parallel, 
so any number of PEs can set barrier bits in the same clock period without 
20 contention. AH PEs are informed that a particular barrier has been reached 

simultaneously, although due to the time multiplexed nature of the inputs to the 
circuits different processors may be at different points in a spin-loop testing the 
barrier bit. 
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Physical Barrier Synchronization 


Circuit Inputs 




First £ 


Second C 


Third C 


Fourth C 
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ofBSRl 


ofBSRl 


Bit 2' 


Bit2 5 


Bit 2' 


Bit 2" 


of BSRO 


of BSRO 


ofBSRl 


ofBSRl 


Bit2 J 


Bit 2 6 


Bit 2'° 


Bit 2 U 


of BSRO 
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ofBSRl 
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Bit 2 J 


Bit2 7 


Bit 2" 


Bit 2 15 


of BSRO 


of BSRO 


ofBSRl 


ofBSRl 



With the preferred radix-4 AND-tree implementation such as that 
shown in FIGURE 5, each level of the tree takes approximately one clock period in 
logic and another 1 to 1.5 clock periods in wire travel time to accomplish. This 
assumption allows the latency for a barrier between 1024 processors to be estimated 
and compared with known latencies for software barrier techniques. 

Table 8 illustrates the number of barrier bits and the estimated number 
of clock periods at each level of a radix-4 tree such as the one shown in FIGURE 5 
connecting 1024 PEs. The radix-four tree reduces the barrier bit count by a factor of 
four in each level. 



WO 95/16236 



PCT/US94/14067 



23 

Tabte 8- 

Barrier bit logical product tree levels 

Number of 

barrkr bits Radivfowr logfcal product tree lml 

5 1024 Level one, clock period 0 

256 Level two, clock period 2 

64 Level three, clock period 4 

16 Level four, clock period 6 

4 Level five, clock period 8.5 

(1.5 clock wire) 

.10 1 Level six, clock period 1 1 

(1.5 clock wire) 

From Table 8 it can be seen that eleven clock periods are consumed to 
perform the logical product of the barrier bits from 1024 PEs using a radix-four tree 
with two to two-and-a-half clock periods per level of delay. If it is further assumed 
15 that the necessary fan-out of the final logical product all 1024 PEs was performed 
using a series of one-to-four single-clock fan-outs, another eleven clock periods 
would elapse, bringing the total latency time of a barrier propagation to 22 clock 
periods. 

From this, the projected performance impact of the radix-4 tree 
20 implementation is relatively straight-forward to predict. With a time multiplexed tree 
that is four bits wide, 4 cycles of the tree are required to update the values of all 16 
barrier bits. This is in addition to the approximately 22 clocks of initial latency. 
The best case occurs when the last processor sets a barrier bit at the same time the 
4-bit "slice" that bit is a member of is entered into the logical product network (refer 
25 again to Table 7), adding nothing to the tree latency of 22 clocks. The worst case 
occurs when the last processor just misses the best case, setting a bit one clock 
period after the slice that bit is a member of was entered into the network. When 
this happens, the slice must wait 4 clocks to be entered into the network and another 
22 for the propagation delay of the network. The final result is a minimum time of 
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22 cycles, maximum of 26, and an average of around 24 cycles. The resulting delay 
is about 160 nanoseconds at a 6.6 nanosecond clock period, assuming that the barrier 
network is clocked at the same speed as the processor. This delay is several orders 
of magnitude faster than known software barrier implementations, which can 
5 approach 500 microseconds or higher. This significant advantage of the present 
barrier synchronization mechanism will be well appreciated by those of skill in the 
art. 



Physical Partitions 

10 Note in FIGURES 4 and 5 that the "shape" of the fanout tree closely 

matches the configuration of the AND-tree. In the preferred embodiment, the fan-in 
nodes performing the AND function are physically located on the same integrated 
circuits as the fanout blocks at each level of the tree. An AND gate and fanout 
block pair is called a bypass point. This advantage of location makes possible 

15 programmable subdivision, or physical partitioning, of each barrier synchronization 
network into independent subnetworks by rerouting the output of the AND gate in 
each bypass point. 

FIGURE 8 shows bypass points 302, 304, 306, 308, 310, 312 and 314 
in a simplified radix-2 barrier synchronization circuit. In the preferred embodiment, 

20 the output of the AND gate in each of the bypass points can be redirected so that the 
output of the AND gate connects to the fanout block in that bypass point. For 
example, FIGURE 9 shows the same barrier synchronization circuit as shown in 
FIGURE 8. However, the output of the AND gate in bypass points 306, 308 and 
310 is redirected to the fanout block in those bypass point. This results in a 

25 partitioning of the physical barrier synchronization circuit into three smaller barrier 
synchronization circuits, designated by phantom lines 402, 404 and 406. 

The first circuit designated by phantom line 402 contains a two-level 
AND-tree and two-level fanout tree. This circuit is created by redirecting the output 
of the AND gate in bypass point 316 to the fanout block in bypass point 0. This 

30 smaller circuit operates identically to the barrier synchronization circuit in FIGURE 
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8. However, this circuit receives an input from and sends an output to PEs 0 
through 3 only. 

The second and third circuits designated by phantom lines 404 and 
406, respectively, each contain a one-level AND-tree and fanout tree. These circuits 
5 are created by redirecting the output of the AND gates in bypass points 306 and 308 
to the fanout blocks in bypass points 306 and 308. 

The bypass mechanism just described can be used to create several 
types of barrier partitions. FIGURE 10 shows the resulting partitions when the PEs, 
in a 1024-PE MPP system using a radix-4 AND-tree and fanout tree are partitioned 
10 at each level of a physical barrier synchronization circuit. Each bypass point in level 
1, of a physical barrier synchronization circuit connects to four PEs for a total of 256 
level 1 bypass points: If all 256 of these bypass points have the output of the AND 
gate redirected to the fanout block, the PEs in the system are divided into 256 4-PE 
partitions, as shown in FIGURE 10A. 
15 If all 32 of the level 2 bypass points have the output of the AND gate 

redirected to the fanout block, the PEs in the system are divided into 64 16-PE 
partitions, as shown in FIGURE 10B. 

If all 8 of the level 3 bypass points have the output of the AND gate 
redirected to the fanout block, the PEs in the system are divided into eight 128-PE 
20 partitions, as shown in FIGURE 10C. 

If all 4 of the level 4 bypass points have the output of the AND gate 
redirected to the fanout block, the PEs in the system are divided into four 256-PE 
partitions, as shown in FIGURE 10D. 

If both of the level 5 bypass points have the output of the AND gate 
25 redirected to the fanout block, the PEs in the system are partitioned as one 1024 PE 
partition, as shown in FIGURE 10E. 

The partitions shown in FIGURE 10 are only a subset of those that 
can be achieved using the bypass mechanism of the present barrier invention. 
FIGURE 10 shown only those achieved by redirecting all of the bypass points at 
30 each level. However, any number of bypass points in any number of different levels 
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can be redirected to achieve a wide variety of bypass partitions. Thus, many 
different partitions of different sizes can be created simultaneously, in the manner 
such as that shown in FIGURE 9. The result is a very flexible and scalable barrier 
resource. It shall therefore be appreciated by those of skill in the art that different 
5 partitions within the barrier mechanism can have bypass points enabled at different 
levels, and that sub-partitions of the partitions can have their barrier circuits bypassed 
at different levels, etc., conferring a great degree of flexibility of final partition 
subdivision upon the barrier tree. 

When a bypass point of a barrier synchronization circuit has the output 
10 of the AND gate redirected to the fenout block, the PEs in the barrier partition can 
still use all 16 bits in BSRO 300 and BSR1 301; however, because there are only 
four physical barrier synchronization circuits, creating a barrier partition affects the 
four of the 16 bits in BSRO 300 and BSR1 301 which are input to the barrier 
synchronization circuit For example, if a level 4 bypass point in physical barrier 
15 synchronization circuit 2 has the output of the AND gate redirected to the fanout 
block, the barrier partition contains 256-PEs. Because of this barrier partition, bits 
2\ 2 s , 2'°, and 2 14 of BSRO 300 and BSR1 301 (the bits that are time multiplexed 
into the same physical barrier synchronization circuit as described above with respect 
to Table 7) in each of the 256-PEs that are part of the barrier partition only affect the 
20 256-PEs in the barrier partition. In the 256-PEs, these bits cannot be used for 
system-level barrier or eureka synchronization, but can be used only for 
partition-level barrier or eureka synchronization for the PEs in the partition. This is 
because the other 768 PEs in the 1024-PE system are not part of the partition created 
and thus barrier communication amongst all PEs is not available in that instance. 
25 Note that, unlike barrier partitions created using the BSMI register 

described above, each of the PEs in a partition created when the barrier network is 
bypassed still is able to use all 16 inputs of the BSRO 300 and BSR1 301 registers, 
just as in the case of an undivided network. Each PE has all 16 barrier bits available 
to it because none are being masked to create the partition. This gives the 
30 characteristic of scalability to the barrier resource: each sub-network created through 
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bypassing the barrier circuits is an independently functioning copy of a barrier bit. 
Thus, the same bit in the barrier register can serve many separate partitions. 

A further advantage of partitions created when the network is bypassed 
in a manner such as that shown in FIGURES 9 and 10 is that the latency of a barrier 
5 is reduced as levels of the fan-in/fan-out tree are avoided. The smaller the partition, 
the lower the latency. 



Flexible Barrier Resource 

As is well known by those skilled in the art, various processing tasks 

10 require partitions of particular sizes and shapes to achieve the most efficient 
. performance. By redirecting the output of AND gates in different level bypass 
points, a physical barrier synchronization circuit may be divided into any 
combination of the previously described barrier partitions. However, the number and 
shape of the PEs in the partitions achieved with physical barrier partitioning alone 

15 may not exactly match the size and shape of partition desired. 

The barrier network subdivisions are arranged so that they fall on 
power-of two boundaries coincident with the logical partitions created when the 
preferred 3-D torus interconnect network is subdivided. The 3-D torus interconnect 
network can be divided by two in any dimension, raising the probability of 

20 mismatches between the desired partitioning of the interconnect network and the 
partitioning of the barrier network achievable by means of the network bypass 
mechanism. In these cases, the BSMI register is used in concert with the physical 
barrier partitioning mechanism to accomplish the partitioning along the boundaries 
desired. The operating system can use the BSMI register to disable some of the bits 

25 in BSRO 300 and BSR1 301 for some of the PEs and enable these bits for the other 
PEs in the barrier partition to arrive at almost any desired group of PE partitions. 

For example, it may be desirable to logically divide the 512-PE system 
and the associated 8*8*8 3-D torus network shown in the upper left of FIGURE 1 1 
into eight equal-sized 64-PE partitions with dimensions 4*4*4 shown in the lower 

30 left of FIGURE 1 1 . Because of the radix-4 implementation of the barrier 
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synchronization, the barrier synchronization circuits can only be divided by 4 using 
the bypass mechanism and the division is fixed in the dimensions that are affected. 
For example, the level 5 barrier synchronization circuit subdivision using the physical 
barrier partitioning will split the PEs into four 128-PE partitions each dimensioned 
5 4x4x8 as shown at the top center of FIGURE 1 1 . This is an insufficient level of 
subdivision to match the desired interconnect network partitioning already described. 
However, if the next level of physical barrier network partitioning is activated, it will 
split each partition by 4 again, into sixteen 32-PE partitions each dimensioned 4*2*4 
as shown at the far right of FIGURE 1 1 . This results in too much partitioning to 
10 match the desired group of eight 64-PE partitions. 

To achieve the desired degree of partitioning, the banier 
synchronization circuits are first physically partitioned using the level 5 bypass 
mechanism to achieve four 4*4*8 partitions. The final division of the by-8 
dimension is accomplished using the appropriate bits of the BSMI register. To 
1 5 achieve this result, the appropriate bit of the BSMI register of half of the PEs are set 
to 0, while the remaining half have their corresponding BSMI bit set to 1. This 
results in dividing the partitions in half to arrive at the desired partitioning of that 
shown in the bottom center of FIGURE 1 1, which is a match for the desired 
partitioning shown in the lower left of FIGURE 1 1. 

20 

Bypass Circuit Implementation 

FIGURE 12 shows a block diagram of a hardware implementation for 
a radix-4 bypass circuit. The four inputs from the previous level of bypass circuits, 
or from the BSRO 300 or BSR1 301 registers if the bypass circuit is a level 1 bypass 
25 circuit, are input to four-input AND gate 350. The AND function is performed and 
the result output to the A input to mux 354 and to the B input to mux 356, which 
together form a bypass switch 355. 

The "B* input to mux 354 is hardwired to a logical "1". The A input 
to mux 356 is received from the next level of fan-out in the fan-out tree. 
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For normal barrier synchronization circuit operation, select signal "S" 
is set such that the A input to both mux 354 and mux 356 are selected. If the output 
of AND 350 is to be redirected to fan-out circuit 352, select signal "S" is set such 
that the B inputs to mux 357 and mux 356 are selected. Thus, the result of the AND 
5 350 is selected by mux 356. In this manner, the physical barrier partitioning 
achieved by "short-circuiting" the bypass circuits is accomplished. 

However, if the output of the AND is to be redirected to the fanout 
block, the select line of mux 356 will select such that the AND output transmitted to 
the fanout circuit 352 by mux 356. 

10 

Timing 

Because a physical barrier synchronization circuit may be divided into 
partitions, and because the 16 barrier bits are actually implemented by 
time-multiplexing on 4 physical barrier circuits, the total time for a bit to propagate 

15 through the circuit is not constant. This timing inconsistency may cause a bit to be 
in the wrong position when BSRO 300 or BSR1 301 is read. For example, if the 
timing is not set up correctly for physical barrier synchronization circuit 0, a bit that 
was originally written to the 20 bit location in BSRO 300 may appear in the 2 4 bit 
location of BSRO 300 when BSRO 300 is read (refer again to Table 7). 

20 Because the barrier synchronization circuits are time multiplexed, and 

each bit in a barrier register is updated only every four clock periods, it is necessary 
for the depth of the pipelined fan-in/fan-out tree to be a multiple of four clock 
periods deep regardless of how it is configured or bypassed. When a physical barrier 
synchronization circuit is divided into barrier partitions, the time needed for a bit to 

25 propagate through each physical barrier synchronization circuit is not consistent. 

Since bypassing a level can remove a non-multiple-of-four number of clock periods 
of circuitry, it is necessary to have some programmable delays to adjust the apparent 
depth of the barrier network pipeline to a multiple of four after a bypass is 
implemented. These programmable delays, capable of adding from 0 to 3 clocks of 

30 delay, are located at each PE and skew the signal arriving from the barrier network 
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for each of the 4 physical circuits. The delays are programmed by writing an 8-bit 
memory-mapped register called the Barrier Timing register (BAR_TMG). FIGURE 
13 shows the bit format of the BARJTMG register. This register is organized as 
four groups of two bits, each of which represents a delay from 0 to 3 clocks to be 
5 added to one of the four physical barrier synchronization circuit outputs to correct 

the pipeline depth. 

The BARJTMG register is an 8-bit, write-only, system privileged 
register. The BARJTMG register controls the timing of each physical barrier 
synchronization circuit. Table 9 shows the bit format of the BARJTMG register. 

10 

Table ? 
BARJTMG Bit Format 



Bits 


Name 


2 , -2° 


These bits control the timing for 


barrier 


physical barrier synchronization 




circuit 0. 


2 3 -2 2 


These bits control the timing for 


barrier 


physical barrier synchronization 




circuit 1. 


2 5 -2 4 


These bits control the timing for 


barrier 


physical barrier synchronization 




circuit 2. 


2 7 -2 6 


These bits control the timing for 


barrier 


physical barrier synchronization 




circuit 3. 


2 63 -2 8 


These bits are not used. 



25 As an example, the following procedure sets the BARJTMG timing value for 

physical barrier synchronization circuit 0 in the PEs of a barrier partition. Before 
performing the procedure, Barrier hardware interrupt to the processors should be 
disabled. 
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1 . Write a value of 1 1 1 1 16 to the BSMI register in all of the PEs to 
enable barrier bits 2 12 , 2 8 , 2\ and 2°. 

2. Write a value of 11 1 1 16 to the BSFR in all of the PEs to set barrier 
5 bits 2 12 , 2 8 , 2 4 , and 2° to eureka mode. 

3. Write a value of 1 1 1 1 16 to BSRO 300 and BSR1 301 in all of the PEs 
to start the eureka processes. 

10 4. In one of the PEs write a value of 1 1 10 16 to BSRO 300 to indicate that 

bit 20 has completed a eureka. 

5. In each of the PEs, read the value of BSR1 301 which contains the 
value of all 16 barrier bits and apply a software mask so that the only bits 
15 read are 2 12 , 2 8 , 2 4 , and 2° and the remaining bits are set to 0. The value read 

from BSR1 301 may be 1110 16 , 1101 l6 , 1011 l6 , or 0111 l6 . If the value is not 
1110, 6 , increment bits 2 ! through 2° of the BARJTMG register by 1 and read 
the value of BSR1 301 again. 



20 6. If the value read from BSR1 301 is now 1110, 6 , the timing is set up 

correctly. If the value is not 1 1 10 I6 , increment bits 2 1 through 2° of the 
BARJTMG register by 1 and read the value of BSR1 301 again. This 
process must be repeated until the value read from BSR1 301 is 1110, 6 , but 
should not need to be performed more than three times. This is because if it 

25 is correct, the value need not be incremented at all, and if it is not correct, 

incrementing by three will cause the BSR1 301 to run through all four 
possible values. 
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This procedure may be used to set the timing for any of the physical 
hairier synchronization circuits in a barrier partition. Table 10 lists the barrier bits 
affected and the write pattern for each physical barrier synchronization circuit 



5 Tabk 10 

Timing Procedure Values 

0 2 l \ 2\ 2\ and 2° 1110 l6 

1 2 n , 2\ 2\ and 2* 2220 16 

2 2 H , 2 10 , 2 6 , and 2 2 4440 l6 
10 3 2)\ 2 n , 2 7 , and 2 3 8880 I6 



Barrier Synchronization Register Implementation 

A single bit of the barrier registers BSRO 300 or BSR1 301, BMSI 
and BSFR associated with a PE is illustrated in FIGURE 14. As stated above, all 16 

15 bits in the preferred barrier register function identically. For simplicity of 

illustration, only the circuitry associated with one bit is shown. Also, note that the 
barrier network is not shown, nor is the circuitry used to interface the barrier 
registers to the microprocessor. 

In FIGURE 14, multiplexors are market "MUX" and have a select 

20 input "S M and a pair of data inputs "1" and "0" that are gated to the output depending 
on the state of the M S" input. Latches have a data input "D" and a clocked output 
"Q M and may have an enable input "E" that controls when new data is to be captured. 
If a latch does not have an "E" input, then new data is captured every clock. 

The three principle barrier registers are identifiable as the mask latch 

25 802 (for BSMI), the function latch 804, and the barrier register itself, latch 808 (for 
BSRO 300 or BSR1 301) . Data from the microprocessor can be entered into any of 
these latches via the processor writing to special memory-mapped addresses. Control 
hardware decodes the address to generate one of the control signals LD_MSK, 
LD_FCN, or LD^BAR. LD_MSK enables data to be entered the mask latch 802, 

30 LD_FCN enables data to be entered the function latch 804, and LD_B AR controls 
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the input multiplexor 806 that gates data into barrier register latch 808 every clock 
period. Data in the mask 802 and function 804 latches is held in the absence of the 
enable signals, while the barrier latch hold path is through multiplexors 810 and 806. 

If mask latch 802 has been set to a 0 (disabling the barrier bit), data is 

5 inhibited from entering the barrier register latch 808 by forcing a constant 1 via OR 
gate 805. This forces a constant 1 into the barrier network as well, allowing other 
selected bits at other PEs to function normally. A 0 in the mask latch 802 also 
forces a 1 into the receive latch 812 through OR gate 811. This disables the local 
processor from "eavesdropping" on the barrier activities of other PEs while 

10 deselected and prevents spurious barrier interrupts from deselected barrier bits. 

Function latch 804 controls multiplexors 814 and 810. If the function 
latch 804 is set to 0 (barrier mode), mux 814 delivers the current state of the barrier 
register while mux 810 causes the barrier register latch to remain set if it is currently 
set as long as the incoming barrier state is still a zero (i.e., the barrier has not been 

15 satisfied). When the incoming barrier state switches to a 1, indicating that all PEs 
have set their barrier bits, then the barrier register hold path is broken by AND gate 
816 and barrier register 808 reverts to a 0 until it is set again by the processor. 
Thus, the barrier mode is a synchronous and immediately reusable mode. 

If function latch 804 is set to a 1 (eureka mode), mux 814 

20 continuously gates the incoming state of the barrier tree to the processor through 811 
and 812 and, while mux 810 causes the barrier register latch to remain at whatever 
state, 0 or 1, that the processor sets it to. Thus, the eureka mode is asynchronous: 
any changes to the barrier registers flow through the barrier tree and are directly 
sampled by the processors, allowing the AND-tree of the barrier synchronization 

25 circuit to be used as an OR-tree for eureka barriers. 

Although specific embodiments have been illustrated and described 
herein for purposes description of the preferred embodiment, it will be appreciated by 
those of ordinary skill in the art that a wide variety of alternate and/or equivalent 
implementations calculated to achieve the same purposes may be substituted for the 

30 specific embodiment shown and described without departing from the scope of the 
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present invention. Those of skill in the electrical and computer arts will readily 
appreciate that the present invention may be implemented in a very wide variety of 
embodiments. This application is intended to cover any adaptations or variations of 
the preferred embodiment discussed herein. Therefore, it is manifestly intended that 
5 this invention be limited only by the claims and the equivalents thereof. 
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What Is ctoimsfl Is? 

1. A computer system (100) comprising: 

a plurality of processing elements (PEs) (200) including a first processing 
5 element (PE) (200), wherein said first PE (200) comprises: 

a barrier synchronization register one (BSR1) (301), said BSR1 
(301) including a first BSR bit (340) and a second BSR bit (342), 
wherein said first BSR bit (340) and said second BSR bit (342) are set 
when said first PE (200) reaches a first barrier point and a second 
10 barrier point, respectively; 

a first BSR output (401); and 

a coupler which couples a value representative of said first 
BSR bit (340) to said first BSR output (401); and 
a barrier synchronization circuit (BSC) (400) having a first physical barrier 
15 synchronization circuit (PBSC) (402), wherein said PBSC (402) comprises a plurality 
of bypass points including a first bypass point (302) and a second bypass point (310), 
wherein each of said bypass points comprise: 

a fanin gate (330) having a plurality of fanin inputs and a fanin 
output, wherein said fanin output generates an indication of whether 
20 said fanin inputs are all set; 

a fanout circuit (331) having a fanout input and a fanout 
output; and 

a bypass switch (355) for switchably coupling said fanin output 
to said fanout input; 

25 wherein one of the fanin inputs of said first bypass point (302) is coupled to 

said first BSR output (401), wherein one of the fanin inputs of said second bypass 
point (310) is coupled to the fanin output of said first bypass point (302), wherein 
the fanout input of said second bypass point (310) is coupled to the fanin output of 
said second bypass point (310), wherein the fanout input of said first bypass point 
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(302) is coupled to the fanout output of said second bypass point (310), and wherein 
the fanout output of said first bypass point (302) is coupled to said first PE (200). 



2. The computer system according to claim 2, wherein 

5 said coupler comprises a time multiplexor (TM), wherein said TM time- 

multiplexes said first BSR bit (340) and said second BSR bit (342) on said first BSR 
output (401). 

3. The computer system according to claim 2, wherein said BSC 400 further 
10 comprises a second PBSC (402), wherein said second PBSC (402) comprises: 

a third bypass point (302) and a fourth bypass point (310), wherein each of 

said bypass points comprise: 

a fanin gate (330) having a plurality of fanin inputs and a fanin 
output which generates an indication of whether said fanin inputs are 
15 all set; 

a fanout circuit (331) having a fanout input and a fanout 
output; and 

a bypass switch (355) for switchably coupling said fanin output 
to said fanout input; 

20 wherein said first PE (200) further comprises a second BSR output (401); 

wherein said coupler further couples a value representative of said second 
BSR bit (342) to said second BSR output (401); and 

wherein one of the fanin inputs of said third bypass point (302) is coupled to 
said second BSR output (401), wherein one of the fanin inputs of said fourth bypass 

25 point (310) is coupled to the fanin output of said third bypass point (302), wherein 
the fanout input of said fourth bypass point (310) is coupled to the fanin output of 
said fourth bypass point (310), wherein the fanout input of said third bypass point 
(302) is coupled to the fanout output of said fourth bypass point (310), and wherein 
the fanout output of said third bypass point (302) is coupled to said first PE (200). 



30 
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4. The computer system according to claim 2, wherein said BSC 400 further 
comprises a second PBSC (402), wherein said second PBSC (402) comprises: 

a third bypass point (302) and a fourth bypass point (310), wherein each of 
said bypass points comprise: 
5 a fanin gate (330) having a plurality of fanin inputs and a fanin 

output which generates an indication of whether said fanin inputs are 
all set; 

a fanout circuit (331) having a fanout input and a fanout 
output; and 

10 a bypass switch (355) for switchably coupling said fanin output 

to said fanout input; 
wherein said BSR1 (301) further includes a third BSR bit (341) and a fourth 
BSR bit (343) which are set when said first PE (200) reaches a third and a fourth 
barrier point, respectively; 

15 wherein said first PE (200) further comprises a second BSR output (401); 

wherein said coupler comprises a time multiplexor (TM), wherein said TM 
time-multiplexes said first BSR bit (340) and said second BSR bit (342) on said first 
BSR output (401), and wherein said TM time-multiplexes said third BSR bit (341) 
and said fourth BSR bit (343) on said second BSR output (401); and 

20 wherein one of the fanin inputs of said third bypass point (302) is coupled to 

said second BSR output (401), wherein one of the fanin inputs of said fourth bypass 
point (310) is coupled to the fanin output of said third bypass point (302), wherein 
the fanout input of said fourth bypass point (310) is coupled to the fanin output of 
said fourth bypass point (310), wherein the fanout input of said third bypass point 

25 (302) is coupled to the fanout output of said fourth bypass point (310), wherein the 
fanout output of said third bypass point (302) is coupled to said first PE (200). 

5. A computer system according to claim 2, wherein the fanout output of said 
first bypass point (302) generates an interrupt to said first PE (200). 
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6. A computer system according to claim 2, wherein the fanout output of said 
first bypass point (302) controls a value which is polled by said first PE (200). 

7. A method for barrier synchronization on a computer system, said computer 
5 system having a plurality of processing elements (PEs) (200) including a first PE 

(200), and a physical barrier synchronization circuit (402) including a plurality of 
bypass points (302, 310) each having a fanin gate (330) and a fanout circuit (331), 
wherein said bypass points are arranged in a tree having a plurality of levels, each of 
said plurality of PEs (200) including a barrier synchronization register one (BSR1) 
10 (301) and a first BSR output (401), each said BSR1 (301) having a first BSR bit 
(340) set when its associated PE (200) reaches a first barrier point, the method 
comprising the steps of: 

outputting a value representative of the first BSR bit (340) on the first BSR 
output (401) of each of said PEs 200; 
15 fanning-in a value from the first BSR outputs (401) from a first plurality of 

said PEs (200) to generate a first barrier-completion signal (403); 

selectively bypassing said first barrier completion signal (403) at one of the 
plurality of levels from the fanin gate (330) to the fanout circuit (331) in order to 
partition the tree; and 

20 fanning-out said first barrier completion signal (403) to each of said first 

plurality of PEs (200). 

8. A method according to claim 8, wherein each said BSR1 (301) further 
includes a second BSR bit (342) set when its associated PE (200) reaches a second 

25 barrier point, wherein: 

the step of outputting comprises the step of time multiplexing said first BSR 
bit (340) and said second BSR bit (342). 

9. A method according to claim 8, wherein each said BSR1 (301) further 

30 includes a second (342), third (341), and fourth BSR bit (343) set when its associated 



WO 95/16236 



PCT/US94/14067 



39 

PE (200) reaches a second, third and fourth barrier point, respectively, and a second 
BSR output (401), wherein the step of outputting comprises the steps of: 

time multiplexing said first BSR bit (340) and said second BSR bit (342) to 
said first BSR output (401); and 
5 time multiplexing said third BSR bit (341) and said fourth BSR bit (343) to 

said second BSR output (401). 



1 0. A system for processor barrier synchronization of a plurality of processing 
elements (PEs) (200) in a distributed processing computer (100), comprising: 
10 a barrier detection program means operative in each PE (200) to determine 

. when each individual PE (200) has reached a processor barrier; 

a plurality of boolean barrier synchronization register (BSR) outputs (401) 
from each PE (200), each output (401) controlled by the barrier detection program 
means, indicating when a processor barrier was reached by that PE (200); 
15 a plurality of physical barrier synchronization circuits (402) for detecting 

which PEs (200) have reached a particular processor barrier, comprising: 

a plurality of boolean logical AND gates (330) connected to 
corresponding BSR outputs (401) of the plurality of PEs (200) to 
indicate when all of the PEs (200) reach a processor barrier; 
20 an ALLDONE signal output (403) from the plurality of AND 

gates for providing an output indicating that all of the PEs (200) have 
reached a particular processor barrier; and 

a plurality of logical fanout devices (331), with inputs 
connected to the ALLDONE signal line and outputs connected to the 
25 multiplicity of PEs (200) to indicate to every PE (200) in the system 

that the particular processor barrier was reached; and 
a plurality of barrier synchronization register bits (340, 341) for storing the 
status of each barrier synchronization process. 
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Fig. 5 




SUBSTITUTE SHEET (RULE 26) 



WO 95/16236 



6/14 



PCTYUS94/14067 



to 
6 



to 

CD 



o 

DC 

</> 

CD 



in 
'cm 



o 
o 
o 
o 
o 
o 
o 
o 
o 

o" 
o 



a: 
to 

00 



3 

to 

ZD 
Z 



s 

tn 
z 



to 



o 

CO 
CO 



in 



o 

CM 

.CO 
CO 



O 



CO 
CO 



K o: o 
cn r> R 

°£ 5 
°P o 

fc£5 



CD 



SG 



^fh z 
a: a: o 

Jfi B 
muJ § 

op § 



to 

c/> 



X 

o 



WO 95/16236 



7/14 



FCT/US94/14067 




WO 95/16236 



8/ 1 4 



PCT/US94/14067 




WO 95/16236 



PCT/US94/14067 



9/ 1 4 




WO 95/16236 



PCT/US94/14067 



10/14 




SUBSTITUTE SHEET (RULE 26) 



WO 95/16236 



PCT7US94/14067 




SUBSTITUTE SHEET (RULE 26) 



WO 95/16236 



PCTAJS94/14067 



,12/14 




/ 


/ 


/ 




/ 














/ 




/ 


/ 




/ 









CO 

co a: 

< UJ 

Ms 



m < CO ^ 



CO 

UJ 

CL 

CM 



a: 
uj 

o: 
or 



(0 
CO 

> 

CD 



y co 

a: d 
a: Ij 
< Q - 

CD 



(/> 
UJ 
CL 

s 



CO 



UJ * z 

§il§ 

"J a C 5 
u z a 




SUBSTITUTE SHEET (RULE 26) 



WO 95/16236 



PCT/US94/14067 



13/14 



Fig. 12 
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